1. Field of the Invention
The present invention generally relates to methods of testing processes, and in particular to methods of testing large-scale processes involving interaction between multiple systems. The present invention also relates to corresponding systems and apparatuses for testing processes.
2. Related Art
In the normal course of business of many organizations, it is necessary to perform large-scale processes that involve one system interacting with other systems. In addition, in view of, for example, competitive concerns or technological advances, it is often desired to develop (improve or update) such processes in order to increase efficiency, to adapt to changes in, e.g., conditions of operation or input/output requirements, or for other reasons. Such development of a process not only involves modifying the process but also requires testing of the modified process to validate it. Validating the process may here be taken to mean confirming that the modified process still performs its intended purpose, e.g., still achieves its intended results or outcomes in response to known inputs.
Validation of the modified process to a satisfactory level of certainty or reliability is necessary prior to putting the modified process into real-world operation. For, while testing of the process (e.g., every test run) costs time and money, failure of the modified process during real-world operation causes losses which, although possibly indirect or consequential, are likely to be of much greater magnitude. For example, failure of a process during real-world operation may result in loss of customers due to dissatisfaction caused by the failure, or loss of business reputation and/or good will, due to publicization of the failure or of its effects on the organization itself or on others interacting in some fashion with the organization. Such losses attributable to the failure effectively amount to loss of future sales, patronage, support or the like and to increased difficulty in attracting alternative sources of the same to replace such losses. Thus, it is often the case that the real (if not always immediately apparent) costs of failure of the modified process during real-world operation greatly exceed the marginal costs of such further testing (and developing) of the modified process as would have been necessary to prevent the failure (by achieving a higher degree of validation (higher level of reliability) of the modified process).
Accordingly, the need for testing processes, in the course of their development, is both great and ever present. While the cost of real-world failure may dwarf the cost of testing, still the cost of testing is significant. Not only are there the relatively quantifiable costs of an organization's simply physically running the tests (including the costs of creating and maintaining the test environment). But also—as our concern here is particularly with processes involving interaction between the organization's system used to execute the process and other systems used in executing the process—there are the same kinds of costs imposed on the other systems.
(Although the primary concern herein is the testing of processes, the testing organization's system used to execute the process may be referred to as the ‘system under test’ (i.e., the system being tested), inasmuch as execution of the process on the tester's system is being tested. While the other, interfacing systems are used in executing the process, execution of the process on those systems is not being tested. From the tester's point of view, the execution of the process on/by the other systems is intended to be a constant with respect to the tester's testing and development of the process. In the ensuing discussion, it will often be more convenient to speak of the system, rather than the process, being tested.)
Among the other systems with which the system under test interacts, there may be other systems of the testing organization itself (“internal” systems) and other systems of other entities (“external” systems). While the immediately apparent costs (establishing and maintaining the test environment and running the tests) to systems external to the organization may seem to be borne by the owners of those external systems, these costs may ultimately rebound to the detriment of the organization itself. For example, if the organization requires the performance of an amount of testing deemed by the owner of an external system to be inordinate or excessive in terms of the costs (time, resources, money) imposed on the owner, the owner may balk at or outright refuse continued testing by the organization. In such case, the organization could also lose reputation/good will on the part of the owner, whose services the organization may require. The owner may choose to work with a competitor of the organization that makes lesser testing demands on the owner, rather than working with the organization. The organization may be forced to turn to a different provider of the external system in question, which may charge more for its services. Thus, it is necessary to reduce not only the costs of the testing at the side of the testing organization, but also the costs that the testing imposes on the external systems with which the system under test interacts.
In addition to the costs of testing imposed on each party (namely, the organization performing the testing, on the one hand, and the owner of the other system, on the other hand) individually, there are costs imposed collectively on both parties. Thus, whenever testing is performed involving interaction between the organization's system and another system, there is required a great deal of communication between the organization and the entity administering the other system—planning, coordinating and jointly performing and evaluating the testing. In some cases, a single step in a process may require the organization's system to interact with multiple other systems. The costs of testing a modification of such a step in the process may be greater out of proportion to the number of systems involved, on account of the synergies created, e.g., by the many permutations of interrelationships between the parties involved.
Another problem that arises in testing processes involving systems that interact with other systems is that the other systems may not be consistent in their responses. For example, the same action (e.g., providing a given input to a given external system) performed by the system under test at two different times may result in two different responses from the external system, even though the external system is designed to give the same response to the same input from the system under test, irrespective of the difference in time. The reason for the external system's giving the different response could be that the external system is defective. (In this case, the response is incorrect.) Alternatively, the reason could be that, between the two different times, some factor—outside of the control of the testing organization (e.g., a factor in the external system or a factor in the outside world that affects something in the external system)—changed, legitimately causing the external system to give a different response to the same input from the system under test. (In this case, the response is correct, but is not controlled, and may not be expected, by the testing organization.)
To be sure, in the case where the other system is not defective but simply is affected by some factor such that it legitimately gives a different response to the second performance, it may be possible for the tester to determine whether the second response of the other system is legitimate in view of the factor, and in some cases it may be possible for the tester to use that knowledge so as to be able to make use of the two test runs as valid test runs. However, even in such a case, it will cost the tester time and resources to acquire and apply this knowledge. The costs of acquiring and applying the knowledge may equal or exceed the cost of repeating the test runs. Thus, at the least, the testing of the process is disrupted and made slower and more costly whenever the other system provides inconsistent responses. It is also possible to envision a case in which the results of the testing are degraded because invalid results (based on inconsistent responses from the other systems) were unwittingly included by the tester as valid results. Of course, such a case poses a significant danger of failure of the process (e.g., giving wrong results) during real-world operation.
The above example may be used to shed light on what is involved in testing a process in general. In that example, the system under test performed the same action at two different times. Since the difference in time was irrelevant to how the other system was designed to respond to the action, the other system was expected to give the same response to both performances of the action. More generally, the responses given by an interfacing system to two performances by the system under test should be the same in all respects not relevant to the differences between the performances; the responses should differ only in the way they are expected to differ based on the differences between the performances. Thus, whereas a process is developed by making a change to it that is designed to have a specific effect, the modified process is tested by running it and determining if the actual effects of the change (the differences between the post-modification run and the pre-modification run) are what would be expected based on the change. This is necessarily the nature or logic of testing a process.
This point may be made in less abstract terms, as follows. The developer of the process inserts into the process a change that he has designed to have a specific effect. Thus, the developer knows what effect the change should have (e.g., nature and magnitude of effect) and knows the scope the effect should have (e.g., which parts of the process the effect should affect). Often, the change is expected to have no effect on a large portion of the process. After the change is made, the developer tests the modified process by executing it and observing the results. If upon execution of the modified process, the developer notices effects different in nature, magnitude or scope from that which was expected, then the modified process has failed the test. The change made to the process resulted in unexpected effects. The effects caused by the change will have to be analyzed until they are understood by the developer. When the developer comes to understand why the change caused the effects it did, he will from then on expect the change to cause those effects. When the test run of the modified process results in the effects that are expected in view of the change that was made, the process has passed the test.
In sum, in view of the nature of testing a process and the difficulties attendant thereto, especially if the process includes many operations (e.g., steps) and requires interaction with many systems, there is a need for reducing the costs (in money, time, resources, etc.) of testing, both the costs imposed on the tester and the costs imposed on the owners of other systems with which the system under test is designed to interact in the normal course of its operation. In addition, there is a need for reducing reliance on such other systems altogether, in order to minimize the occurrence of invalid test results and thereby increase the productivity of the testing process.